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Outcome evaluation assesses the results or benefits 
health services received by clients or communities by 
descriptive da^a on the mental health status of clients at 
points in time, 1^ a^ ds . clinici ans and managers in planning 
and managing clinical services. A mental health" center 
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should establish goal -ori ented prcqraro planning, information systems, 
and quality assurance proqrams before it attempts Client outcome 
evaluation. Outcome '5tudies are still in the developmental stages and 
should b'e conduc+ed when ■♦•here are clearly defined needs, available 
resources, and clinicaT and miinaqerial aqreement regarding the' uses 
of the resultant da ts . Seven outcome evaluation methods are described 
and referenced: (1) .level of f unc-^-ioninq scales; (2) irultidimensional 
ratings: (3) 5CL-90 Symptom Checklist;' ( qoal attainment scaling; 
(5) client satipf action follow-up questionnaire: (6) community 
satisfaction studies: and (7) consultation and educatlan cu^c^pmes. 
Client outcome studies can be pontinuous and provide* on -going* 
fOTdback, or they can be terminal and provide information about a, 
particul-ar service or group of client's. These studies assist J.n 
forming policy, > deyfelopinq proqrams, and just if ying .programs. 
(Auikhor/MLT) 
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The South(>rn Rofjonal Kclucat Ion Board was awarded*a grant (Mental Health 
Tratnlnj^. Crant No. M -T I S-MII I AS^Ol) In late 1976 from the State . Manpower and 
Development Branch of the Na,tlonal InsStltute of Mental Health. The Project 
was to* develoj) publications and conduct workshops to assist mental health 
centers In Improving tho'lr management practices and their program activities 
through the use of practical program evaluation. A series of publications 
and workshops is beUng developed through the combined efforts . of the Board's 
staff and task force participants. Topic areas include : 

o The Administrative Uses of Program Rvaluatiory^ 

o Use of Information Systems for Monitoring Mental Health Programs 
,o Linking Needs Assessment to program Planning and Management 
o Ouallty Assurance in Mental Heajth Centers ' ' ^ 

o Cllen't Outcome Evaluation in Mental Health Centers < 
o Improving Staff Productivity in Mental Health Centers 

♦ 

The selection of these topics was based on the preferences expressed In a 
survey of mental health centers and clinics '4n .the 14 states served by the ^ 
Southern Regional Edut^atlon Board. 

Client Outcome Evaluation In Mental Health Centers describes outcome 
evaluation methods » and explores sofne of th'e ways that studies can b^ con^ 
ducted and used. This publication is based on the recommendations of people 
Ir) mental health centers and state mental health agenclies. We thank all of 
them for their willingness to share their knowledge ^nd experiences with us. 
We assume responsibility fojp^the content of this report, including any 
misimderstandlngs resulting from the translation of Ideas. 



Janet F. Despard^ Project Director 
Improving Mental Health Centers and ' 
Mental Health Planning Project 

Harold L. McPheeters 

Director, Commission on Mental Health 
and Human S^ylces 




TASK FORCE MENTOERS 



/hi Hp Bark.lev 
Director of Evaluation 
Department of Mental Health/ 

Mental Retardat.lon * 
Nashville, Tennessee' 

•Jack Franklin ' ^ 

Texas Department of Mental Health/ 

Mental Retardation 
Austin, Texas 

William Hargreaves 
Langley Porter Institute - , 
University of California 
San Francisco, California 



SOUTHERN REGIONAL EDUCATION 
BOARD STAFF 

Harold L. McPheeters 
Director 

Commission on Mental Health and 
Human Services 

Janet F- Despard 
Project Director 

Improving Mental Health Centers and 
Merttal Healtjh Planning Project, 



Frederick Newman 
Eastern Pennsylvania PsychiAjtrxc 

Institute 
Philadelphia, Pennsylvania 



\ 



Jerome S. Nichols 

Mental Health S^r\;ilces of 

Roanoke Valley 
Roanoke, Virginia 



t 



r 



Iv 



ERIC** 



•WllAT 'ls'*r)llT(:()MK KVAI.UATION? 



( 

There appears to be a limited a)>r.eemont- in the mental health Field on 

the meaning of the term "otitconja evaluation," A general definition f^und in 

the SRRB publication, IkM^LnLtLon of Terms i n Mdntal Health, Alcohol Ab use, 

Drug AUise, and M ental Re targiat ion , defines program outcome evaluation as: 

"The effects achieved for a target population by a j:>rograni . " This definition 

is expanded with the rationale that: 

^ There has been much attention given to the service delivered by. 
staff persons, but less attention given to the benefits or results 
to clients or communities. Out come 're f ers to the changes or 
benefits brought about in clients or communities as a result of 
the services delivered. Clutcomes are seen *as effects, changes, or 
impacts on recipients, ■ . ^ 

This general definition is probably acceptable to most people. Problems 
arise, however, when a more specific description is recjuiifed to explain why ' 
outcome evaluation is done, how outcomes are measured, and what usas are made 
of findings. . ^ 

V 

The purpose of outcome evaluation is to find out whether clients are 
being helped by, the mental health services they receive. Federal and state 
administrative agencies and legislative bodies are now exerting pressure on ^ 

mental .health centers to document the effectiveness of their services to 

\ , • 

clients in the cfommunity- BiJt the expectations of outside agencies are not 

the only reasons for evaluating program outcomes. The information drawn from 



I 

outcome ova 1 uat ion. I s also usotul to'tho admin I Ht rat I vc and clinical staff In 

a center* Some of the questions that outcome evaluation can answer are: 

o Is mental health tre^t^inent. heli)lnR clients?^ 

o How much does, treatment helj) clients? / 

/ 

o What kinds of clients benefit more from a particular kind of 

treatment? * ^ ^ 

o Is one service modality more effective than ^another? 
o Are clients satisfied with th» servic^sthat they xfecteive? 
o Is the community satisfied with the center's services? 
These quest ions| can be answered by using evaluation or research 
apf^oaches. 

O utcome Evaluation 
.Outcome evaluation uses descriptive data on Client' status at different 
points in time to come to a conclusion about the influence of treatment on 
clients. Scientific rigor is not nearly as Important as producing findings 
that are appropriate for decieion making fend useful for educating management 

and cl in ical -staff . This' more pragmatic approacfi often does not produce 

/ ' 
Information!^ that can be generalized to other settings, but methodologies 

shoi(ld be used that will give the most precise answek^osslble , given the 

constraints of time, Resources, and planned uses of the data. Findings from 

outcome evaluation are* used to trigger action by clinicians and managers in 

assigning clients to particular therapies, allocating resources to programs, 

and; identifying inserllce training needs. They also may be usdl by manager^ 

to restructure existing programs. 
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(Outcome l-.va I uat ton 2^J:^^^'^^'J'J^ " * 

Kvc'iliiat ion research is a form of appl IccU sc lopt 1 f tc research which ^ 
measures cLiangu in client status over time to draw causal inferences about ^ 

A 

the etfect or treatment on clients. A research cleslp^n with random selection 
of clients' or equivalent e<^ntrols is used so that the relative effectiveness 
of two treatment ai^proaches (\an he comi^ared. This kind of study Involves the 
Investigation of events under scientific rules of evidence to generalize 
kaowledge. Research resul-t^ are used In developing alternative treatment 
approaches within programs In mental health centers. 

Evaluative research is feasible when centers have the research talent 
and other resources available. Otjierwise, it is not sensible for agencies 
with limited resources to spend service delivery dollars on rigo^rous scien- 
tific apiii'oaches. This does not imply that centers should not look for ways 
of Improving treatment api^roaches. It does' suggest that most centers should^ 
employ methods of mQasuring client outcomes that will provide practical and 
tlmelv feedback on clinical activities before they attempt mpre sophisticated 
studies. . ' ' ' 

Mention ^hould also be made about comparative studies of the effective- 
ness of various' mental health programs^^^t recommended that comparative 
studle\s are best addressed hy^state and federal agencies because of the costs 
and the problems of compatible data across programs. The exception would be 
when a number 6f centers pool their resources and data to do cpmparative 
studies, ofte^ with suppor't antl technical assistance from state mental health 
agencies. ^ ^ 



OlITCONm KVALIIAT ION Mt'THODS ^ ' ^ 

A niihiber of methods cr\n be used to evaluate outcomes. These metho'ds 
differ In the dimensions of outcomes that are measured. Each method l^as 
advantages and drawbacks related to its specificity, cost, and potential uses, 
Seven different mqthods are briefly describe^ and referenced. 

Level of Functioning Scales 

The client's overall level of functioning Is measured from the perspec- 
tive of the clinician. These global assessment scales are a ^orthgnd. way of 
indicating, a number of factors related to client status. -They mu^t be backed 
up by a good clinical record to clarify what these factors are. 

It is suggested that a broad range of scale points ~ up to 100 with 

10 point levels w:fciin the scale — is more desirable than scales with 5 or 

> * 

10 scale points because the scales with broader range are more sei>sltlve Iri 
discriminating changes in client status. The Indicators produced on broader 
scales are far more useful In quality assurance procedures bacause the 
information they yield is more sensitive.^ 
Re f erences : 

Carter, Dale R. ; Newman, Frederick L.^ A Client-Oriented Systeri of 
Mental Health Service Delivery and Program Management: A Workbook 
and Guide . Department of Health, Education, and Welfare Publication 
No. (ADM) 76-307. WaWington , D. C: Superintendent of Documents, 
U. S. Government Printing Office, 1976. , \J 

F.ndicott, Jean; Spitzer, Robert; Fleiss , Joseph; and Cohen, Jacob. 
Th e Global Assessment Scale; A Procedure for Measuring Overall 
Severity of Psychiatric Disturbance , Unpublished manuscript, March 1975. 
Copies available- upon request from New York State Psychiatric Institute, 
■'\ 122 West 168th Streejt, New York, New York 10032 (Dr. .Endicott). 



Mult Ld lmens ioT\al Ratinjj^s ^ ' 

Pre- ^ind post -mul 1 1 cl Imons lona 1 ratlnf>s of symptoms and personal ^ social » 

^and community adjustment are' avh 1 1 abl e to centers that wish to use them. It 

Is recommencfed that centers carefully choose the Instrument that best suits 

their needs in providing answers tdk the questions which prompted the outcome 

♦ 

study. Furthermore, centers arc ur>>cd to use available scales Instead of ^ 
trying to develop their own- Some vsuggested measures and references are: 

The SCL-90 Syiiytom Checklist is a client self-report form consisting of 
90 items .covering common complaints of psychiatric outpatients. . Because it 
takes little time to administer and score, this checklist can be used on a,, 
routine basi^ with all clients. It can be used to explore the symptomatology 
and progress particular client groups. Its llmitat ions \atG that it ^ 
includes no measures of intlfcrpersonal or social adjustment and, because it is 
confined to a standa^rd set 6f syffiptoms, it does v\pt show unusual or unique 
symptoms . . . - - |^ 

References: ^ ^ 

Dorogatis, L.R.; Lipman, R.S.} and Covi, L. "SCL-90: An Outpatient 
Psychiatric Rating 5^cale" (preliftii^riary report). Psyghof^harmacology 
Bulletin , 9:13-28, L973. . <^ 

# 

Derogatis, L.R.; Llpman, R.S.; Rickels, K. ; llhlerhuth, E.H.; and 
Covl, I.. ''The Hopkins Symptom Checklist (HSCL) : "A Self-Report 
Svonptom Inventory." Behavioral Science , 19:1-15, 197^^, 

Personal, Social and Community Adjustment Scales measare and compare the 
client's personal, social and community adjustment prior to treatment wlt^ 
adjustment at later, points (eith6r during or after treatment). Although the 
three scales shown below share th^ same measurement objectives, these scales 
differ in the^^l^tails of their content, format. Implementation procedures 
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and costs ♦ Tliese scales tend to be lengthy and expensive to administer and , 

♦ 

analyze. They are mosx;\u Ltable t or^p^e In well— designed , tlme-llmlted 
studies rather than on a rout ine^ has Ls with all clients. The analysis of the 
data yielded by t,hese scales can Identify areas of adjustment in which clients 
need help; can be used to develpp new types of services to meet the treatment 
needs o/ talents; and can help assess the durabi^lity of treatment results 
after termination, 

1. The Katz Adjustment Scale (KAS) consists of five scales containing a 
total of 205 Items: 1) personal , symptoms ; 2) performance of social rolesr 

3) Informant's '^expectat ions*' of the client's social role performance; 

4) performance of leisure time activities; 5) informant's expectations about ^ 
leisure time ac tlvlttu^s . These scales have been most widely used in long- 
term psychiatric hospitals. 

4 

Reference: 

Kat^p, M,M. and Lyerly, S.B* "Methods for Measuring Adjustment and 
Social Behavior In the Community:' Rationale, Description, Discrimi- 
native Validity, and Scale Development.'* '" ^Psychological Reports , 
13:505-535, 1963. , 

2. The Personal Adjustment and Role Skills Scale (PARS) coYitains 40 
Items which measure personal and social adjustment^T ^Separate^ersions of the 
scale are used for males and females* This scale is developed for rating by 
an Informant (e*g., the client's family members) and has beeo used success- 
fully as a mall-out questionnaire* ^ 

Reference: 

Ellsworth, R.B. ''Consumer Feedback in Measuring the Effectiveness of 

Mental Health Programs." In: Guttent^g, M- and Struening, E.L. (Eds.). 

Handbook of Evaluation Research , Bevetrly Hills, California: Sage 

Publications, 1975. » ^ 



']\ TJio Denver Communtty Mental Health fhiest lonnalre consists of ^71 items* 
that fcaaus on personal and social adjustment, contact 'with other human service 
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agencies, and client sa'tyTsf act Ion with services. This questionnaire was 
designed spec: 1 f ica I by for use In mental health centers, primarily. to^>£pllow 
up on^'l4ents afLt;r terminat Ion 'of treatment. It Includes both se^lf-re^orts 
by the client and reports by other informants; it, is usually administered 
through structured Interviews in the client's or other informant's home. . 
Reference: / 

Ciarlo., .l.A. and Reitiman^ J. '*The Denver Community Mental Health '\ 
Questionnaire^j, Development of a Multidimensional Program Evaluation y 
Instrument.", ifnpubllshed paper,' 1974. Available from the Mental 
Health Systems Evaluation Project, 70 West Sixth Avenue, Dem/er, 
Colorado 80204. ^ 

Goal A ttainment Scal ing 

Coal Attainment Scaling is a technique that measures treatment outcome 
by assessing the extent Lo" which clients achieve individualized goalg. Tl/e 
^ecific content of the scales is tailored for each client. A goal attain^ 
ment follow-up is done for each client^ to determine the. level of outcome on 
each sc^le. Outcomes for all clients are calculated through the combiyatlon 
of goal attainment scores/of all clients. It is common for each mental 
health center to modify the^aslc technique to meet itfe own specific needs. 

References: 

Kiresuk, Thomas J. and Sherman , Robert E. "Goal Attainment Stealing : 

A General Method for Ryalu^ating Comprehensive Community Mental 

Health Programs." Community Mental Health Journal , 4:443-453, 1968. ^ 

Kiresuk, Thomas J. '"Coal Attainment Scaling at a County Mental 
Health Service." Evaluation , 1:12-18, 1973. 

1 
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Klrosuk, Thomas J. and l.und, Sander H. "Process and Outcome', 
easurement i;s.i,f,v(.o;n Attainment Scaling."- In; Zasman, Jack and 
nrNi""' 1 V ll£2ar^"LJyalua^ ^ pruR Abuse, 

Books, 19 75. 

• ■ 

.The Automated'TJfl-lnformant (ioal -Or iented -ProRre^ss ' No te, .( ATGON) .servel^ 
maov ot'th^> saoie purposes as «oal ai'talnment scnlinR. It is based on the 
^same underlying assumpbion, but the design and procedures differ; 'ATGON Is 
an automated technique which measures client progress toward p^edeb^rinlned 
i^pals. Assessments from three sources are included:- clients, clinicians, 
and relatives of the^j c 1 ients. 

Reference ; ■ ^ 

* 

« 

Wilson, N.C., and Mun$ower, J.L. "Automated- Evaluation of Goal 

Attainment Ratings." jLospltal and^qmmunitv Psv ch^a^rv, 

26:163-164, 1975. ■ ^ ^ . •• 



Clibn^t S atisfaction F-ollov-up Questionnaire ' ^ 

Assessing the sat is f ac t ion V clients with the process and results of 
treatment has become increasingly common In mental health centers,. In part 
becau.<^ of the emphasls^^^n citizen T^lclpatlon and consumerism In human' ' 
service agencies.^ These questionnaires are relatively Inexpensive tO design 
and implement. .The data that they yield can assist In identifying H^lem^ 
In programs, getting feedback from the community on the value of mental 
health services, and providing information ^n "accessibility" and' ^accept^:'^ 
ability" as required by current mefntal health legislation (.P.L. 94-63). ^ 

It is^sufJ>ested that client satisfaction questionnaires be kept simple 
^nd brief - about 5 to 10 questions. These questionnaires usuall.y include 
items directed to the client's satisfaction with treatment and their overall 



satlsTactlon with t\\i? sorv Ict^^J ot^fered by the center .^N^^^'her conctete 
queBtlons that Hfhy be of Interest to the centfct director can klso be added. 
It Is recommended^ that some c|uestion8 be' i\&kh\ two or three different w^y.s^ 
Re t erence r * 



D enver Co mmun 



Ity Mental H e alth Questlgnna j^g (selected questions) : 

James Clarlo, Shle-^ftng Lin, DouRlas Blfielow, and J^rllyn ffiggerstaff 
Denver .Gen^tJ^l Hospital Mepjtal Hea^Jfth Center >• 



,_^n^^er, Colorado 8020A 

Daniel Larsen, C.< Clifford At^tklsson^ and Willlant A. HargreAyes 
tangley Porter Institute ' > .<^' 

«iJnlversity of .California ' .V 

/San Francisco, ,q|Jlifornia 9A1A3 ' ^ ' ^ . 



Community SatlsFaction Studies / j' • 

' . • ■ ■ ■ / ' / 

Outcome indicators can be derived from pre- and post-measures of change 

in community satisfaction based on key Informant reports (e.g., formal center 

affiliates,^ ancillary i^ervices, referring agencies). Several other methods 

can be used. Actual client movement Into and from the center can be tracked 

* 

and compared to the policy and formal service agreements with other agencies. 

< 

Reference: * . ^ 

- Warhelt, G„.J.; Bell, R.A. ; and Schwab, J.J. Planning for Change! ' v 
Needs A ssegsment Approaches . Rockville , ' Maryland : • NatloAal It^stltute^' 
of Mental Health, n.d. 

Consultation and Education Outcomes 

Consultation and education services In.schools, Industry, nursing homes, 
law enforcement agencies, courts and /ther human service agencies can be 
evaluated by comparing actual activities to stated goals for services If the 



otMUor opohatos uiulor <i }u>a 1 --or IcnUed a|)|)roaclK Basic steps In setting 
r.oals artel evaluating outcomes Include: , - 

1. KstabUslUng an agreement wl(ftti the agency that> defines th/ 
gaalM to be achieved by the service; 



2^ Setting the |«an for tlie provision o.f services; 



3.^ Provld ingi tlie services;. 

; 4- ^ Measuring the outcomes by comparing thfe results to stated goals^ 

Xhe readeV Is also referred to two general resource publications for ^ 
Informatlttn on outcome measures: . ' • ' ^ 

Hngeclorn, H.J.; Beck, K..!!; Neubert, S.F.; Werlin, S.H. A Working 
Manual oF S tmple" Program F.v^iluatlon Techniques for Community Menta 1 
H<^-Tlt:h Centers . Department of Health, Education, and- Welfare 
Publication No. (ADM)' 76-404. Washington, D. C. : Superintendent pf 
Documents, U. S. Government Printing Office, 1976. 

Har'l'reaves, W.A.; Attklsson, C.C.; Sorensen, J.E. Resource Materials 
• t 'ot- Community Mental Health Program Evaluation (2nd edition). 

Department of Health, Kducatlon", and Welfare Publication No. (ADM) 
77-128, Washington, D. C. : Superintendent of Documents, U. S. Government 
Printing Of f ice ,Ul 97 7 . 



Wlien Belectinlg outcome measures, the evaluator ghould look for an 
instrument that ^ appropriate and sensitive to chhnge In client states. 
Other considerations are the reliability of the Instrument, tlhe usefulness of 
the TOeasures to clinicians and program managers, and the costs of .the pro- 

^ • " 

cadijtres. Since the major, ways of measuring client outcomes ol>Cert are not 
correlated to each other, the evaluator .should try to select instruments to 
as.qure that 1) the dimension of client status chosen will answer the 
, questions Being raised;. and 2) the resources needed for Inipleme'htation are 
related to the potential pay-off that the findings will have. • 

\ 
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V^u'latlons In tlie deinograph It: , socioeconomic, and cultural charaqter- 

l.sllcs of communities Influence what Ls^* cons Idered "normal'* belmvior and 

timet Ion Ing In cUtterent communities. Al though this Is not an overriding 

ij factor In tbe se lec; t ion. ol outcome scales, knowledge pf community nofms Is 

lielpful In rating and analyzing client out;coities. Some authors have developed' 
» * 

« 

' outcome techn Iques • that compare client outcomes to community norms to assess • 
-/ tlie effectiveness of treatment programs (Clarlo, the states of Oregon and 
Washington),^ Assessments of the need for services which Include the 
chnracttrlst.}.cs of the community (e.g., age, sex, race/ethnicity, income, Hi» 
education, unemployment » crime) assist^ in identifying the nature of factors 
tliat Influence "normal" functioning and behavior In the cpmmunfty. 

? ■ 

Mental health centers should conduct credible outcom^ studies whether 

they are done on a continuing basis or on a special study basis. The ne^d 

for measures that apply across all treatment modalities reduces the scientific 

rigor of scales that can be used, but reliable instruments -are easily found. 

* It Is advisable to select methods that rate both the clinician's and the 

*^ 

client's perspective^ of treatment results. * ' 

ONG OiNC ACTIVITIES IN -PLACIs 

It Is recommended t^hat centers have the following activities^ in place 
before condCipt Ing oytcpme evaluat ion : ^ 

,1. Goal-oriented planning should take place before conducting out'cbnrfe 
evaluations. Written documexYtat ion of the proposed evaluation procedures 
should be included In the plans* 
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Tl.o cvntvr slun.ld \uwy an ovalua'tor or pcrHdn'wUh-^.valuat ion sklil 



v,;lu) sorvoH as a .m>mlH>r ot the rnanny>ement tcnm, 



I 



i. Tho centorVs adm In i st m t Ion nn.st support .program evaluation and have 
the ability to use the evaluation ilndln^H. • ' " . 

A. OperntLons and clinical procedures should be monitored through an • 
lnfor„,atlon s-vstem (or,nniv..d files from which data can be^eadily accessed) 
tliat includes data on: 

o client movement; . . , . . 

o^staff activity that accounts for at least direct service activities 
and Indirect services, such as consultat ion and education; - 

o c^^ost data Including direct costs, indirect service costs, and 
administrative support costs; 

o clinical records that are up-to-date and include treatment .oals 
• and plans tor each 'client. =*Lmeni: goals . 

5. case management should Include the review of client cases by ' 
clinical supervisors and a quality assurance program in the developmental, 
sta-e. ' • ' * 

f>. There should be mechanisms for feedback to clinicians and to the 
cotpmunity. > 
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. CONDUCT I Nr. OUTCOMK EVALUATION • 

Many centers defer client outcome studies to a l^ter staple in th^lrf 
development for a number of reasong. Other program evaluation, activities, 
such as the development of information systems and cost studlesy have a higher 
priority than outcome evaluation In mo^st centers. Implementation of outcome 
evaluation is often delayed because centers do not consider the potential 
payoff in the uses of outcome data to be high enough to Justify the costs 
lnvolve.d* (The exception is when categorical programs require outcome 
evaluations,) Anothef related factor Is that the state of the art of outcome 
studies is still In thd developmental stage and the potential uses of outcome 
data are not always understood, 

PLANNINC AN OUTCOME EVALUATION STUDY ^ 

The fdllowing questions should be asked In the planning stage of kr\ 
outcome study to aid In selecting an appropriate method: 

i • jr ■ ' 

o Why is the data needed? ) 

• * 

o What specific questions are being asked? 
o Who will use the data? , » 
o How often will the /data be needed? ' ' 

o How does the data collection process flt^lnrto other, ongoing program 
evaluation efforts? 



o Are the needed resourced available? 
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Oirtcomo evaluation should be done when there Is a clearly defined need 
for data, resources are available, and administrators and. clinical staff . 
agree to uBe the H ind Ings . These decisions can be made by the center's 
management team. ' 

fn selecting appropriate Instruments for measuring outcomes, it is 
recommended th^it centers choose from standard measures that are already 
avalnible Ihste-ad of trying to develop their own. If the persons im^olved in 
making this choice are not sure about the instrument that shoui« be used, 
they should call in consultants from state or other agencies for assistance 
in the selection of an appropriate measure. 

' When doing an outcome evaluation f'or^ a categorical program (e.g.., Nation- 
al Instltyte of Drug Abuse? requires Client Oriented Data Acquisition Process), 
it is suggested tlmt the center look at the federal and state reporting re- 
quirements for that program before choosing an instrument, even if the center' 
does not. receive cato^;orlcal funds t'or the service. There are two advantages 
to using the shme methods of measurement required by state and federal 
agencies: 1) outcomes can be compared with state and federal - statistical 

data; and 2) if the program does get funding from these agencies', the center 
, ». « # 

will not have to revise its evaluation procedures. 
IMlMJ-MENTINCi AN OUTCOME ICVALUATION/^TUDY 



Clinical and managerial staff should be educated on the meaning and uses 
o1c outcome evaluation before outcome studies are Implementecl . Additional 
training sessions may be necCssary after implementation tp maintain the 
reliability of ratings and to demonstrate the uses of the evaluation findings, 



Staff should bo elvon feedback reports on "f indlnj>^ per lod Lcally so they can ^ 
use tlie .data* for cllnicUil and overall management purposes. 

' Tlie evaliiator can anticipate anxious reactions from staff whe^n they first 
soe tho results of outcome studies because results are ^eldom as good, as 
expected. 'Training sessions on the meaning and uses of outcome data will 

h'&^Lp-^duce these anxieties. The center administrator should judge when and 

* / • * ** 

how. an/ information is released to others; He may decide to withhold the 
/ * 

r6Xi2j/se of outcome ^iii^rmation until the evaluation system is operating 

smoothly and the reliabil^ity of ratings -is well established. 

Clinicians may "be bothered by the words used in an instrument and may 

feel that identifying clients' behaviors with a number is alien to their way 

of thinking. Others may think ?hat a standard instrument is inappropriate 

for the program. Some suggestions for handling these problems are: 

o Try a standard instrument. If it does not produce the desired 
cooperation, try another one. 

o Do not mnke word changes in a standard instrument initially. 
Instead, have training sessions in which the results of 
lndivldua.1 ratings are discussed and consensus on the meaning 
. of rating categories is developed. 

o After the instrument has been used a while, there are two 
possible approaches to increase rater reliability: 1) an 
interpretation for local usage of scales can be agreed upon 

, without^ modifying 'the instrument; and 2) words in the instrument 
can be changed to fit local usage of the scales. If word usage 
Is changed', notations on these changes should be made in any 
^reports. either approach should include staff training 
rejwirding changes in the scales to increase reliability. , 

' - .■ V ■ ■ 
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TWO W AYS OF l--V AUjATlJ^J^ 

There are two basic approaches to outcome evclluation: /Special' studle 
aq^l 2) continuous datn collection. ' The ■ basic procedures that^'have been out- 
^ lined apply to both kinds of outcome evaluation efforts. 

Special studies. Special outcome studies for evaluation require data 
Vthat are not routinely collected or reported In the information system. . 
Special outcome studies might involve sampling from .continuously collected 
; data for a particular subgroup of clients or conducting a full-scale study of 
client outcomes in a .particular program. These studies may be done period- 
ically or only once, depending on the need. Often the nee^^r^r special 
studies is prompted by problems that appear in thep*^?tflar monitoring reports, 
Clinical supervisors' may reqtiest a special study when planning for a 
modification in an ongoing treatment progr.am. The administrator may need 
outcome data to support the development of a new program or to justify 
exist ing one . ^ • 

i;.^ Continuous Data Collection ^ The cont inuous collect ion of outcome 
^information usually involves the rating of all clients who receive services. 
These ratings are recorded in the clinical records to assist in clinical 
management and in the center's information system to provide easily tabulated' 
^data for evaluation. The minimal level of outcome data that' center^s shoifd 
consider maintaining on a continuous basis is: 

1) The overall level of functioning for every client on 
intake. These ratings should be backed up by progress 
records for each client, 

2) ^ The overall level of functionlnf^ for every client at least 

once at a predetermined^oint in time, whether or not the 
client Is still under treatment* 
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1) A client satisfaction follow-up questionnaire to all 
^ terminate^! clients (can be conducted by mail). 

4) SuiTtlnarics of data on levels of functioning and client 

sat Lsfjict Ion follow-up cj^est ionnaires at regular Intervals^- 
with appropriate feedback to the users of the information. 

4. 

ll^K OF LKVEL OF KUNCTIONINC SCALKS ^ ' 

Several level of functioning scales were suggested in the previous 

section. These scales show the overall status of clients at the time of 

rating. Before and after measures show whether any changes have taken place 

whil^ the clients were under treatment. It is recommended that level of 

functioning ratings be done at every clinical contact' with clients to provide 

more adequate information for decision making by clinicians, supervisors, and 

peer review and utilization review committees. 

Regardless of how often level of functioning ratings are done, these 

indicators should be entered in the ctenter's information system in two • 

places:' in the client's progress record for case documentation purposes and 

in the data collection system with other management data. These may be 

reported qn billing tickets for individual clients along with the client's 

identifier, the services rendered, the code for tlie service provider, and the 

fee charged. In a hand-operated informauW system the aggregation of data 

can be facilitated by using a code to identify whether the rating was pre-> 

treatment, during, or aftet treatment. A staff member should be" assigned 

responsibility for monitor.ing data entries to make sure that the data are 

properly and completely collected. 
♦ 

Level of functioning ^ata should be t^ulated and reported to appropriate 
personnel at' predetermined intervals. In smaller centers it may take as long 

' ^ ; 17 - 



an six months before the data base is large enough to provide evidence of 

; 

status changes by program. Kor these indicators to have meaning, they' must 

be psirtltioned according to specific time periods. Client ratings are 

usually partitioned intd the following groupings: ^ 

,1. Clients who entered before reporting period-; 

* 

)^ 2. Cl lents ^ho entered during reporting period; * ^ ^ ' , \ 

3^ Clients who terminated during reporting period; 
» A. Clients who terminated after reporting period. " |||^ 

A partitioning by client typfe Is also recommended to make t^e outcome 

4 

measures more meaningful. Clients can be divided Into those whose goals are 
directe(jl toward Improvement and those whose goals are prlmar'lly maintenance 
of t^ielr present status, or they may be divided by. k,lncj of probleiti or age 

Routine feedback reports should be provided to clinicians, clinical 
supervisors and- center managers. Gene^-ally outcome data Is aggregated by 
client groups so that as reports proceed up the center's hierarchy, Individual 
clients cannot be identified, 

> CLIENT SATtSf ACTION FOLtOW-UP QUESTIONNAIRES . * ' 

Standard client satisfaction questionnaires are ^va^lable to centers, or 
centers may construct their own. There are several basic tssues that shoyld*. 
be considered when using this method : 

o ^Questionnaires should be short * — ^o^more than 10 questions. 

o Questions "should be concrete. They may include satj.sf act ion with * 
results of the treatment or general satisfaction with and 
^ accessibility of the services offered by the center, , 
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o (Questionnaires can be mailed or. In come cafes, filled out 
by clients at the time treatment Is terminated- 

^ o Client satisfaction .studies are sec4)ndary In importanoe to 
7 other evaluat ion act Ivlt les , such as Jcost hnalyses , but 
because they are relatively Inexpensive to do, they are 
usually worth the effort, , - / . 

( 

COST-pUTCOME STUD IKS , ■ 

Client outcomes* can be merged with cost data to derive cost-outcome data 



which relates the cost of providing service to clients to' the degree of 



^Improvement in the^^^f unction ing while in treatment. These studies are used 
to discove.^'^f actors that contribute to differences in costs and client 
outcomes and to help decide about the best ways to deliver services -to client 

' groups, : " 



us INC. OUTCOME RVAIAIATION 

Mental health centtn^s^se outcome evaluation to help in making decisions 
about clinical mancTgement and program planning. The data may also be used to 
support requests 'for allocation of resources. 

Clin le a l Management . - , 

When outcome/ ratings on clients are linked with treatment plans, 

,cllnician>e get an objective view of their work with clients to help them make 

better decisions in managing their case work. They are able tq 'Identify 

those cases In which a treatment approach Is particularly effective and those 

In which they are "spinning their wheels." Staff are also better able to 

make decisions ^on terminations of treatment and transfers of clients. 

Clinicians usually do not like the idea of having the effects of their 

work evaluated. However, if it can be demonstrated that assessing outcomes 

can help the^m.in providing better care to clients, they are less resistapt^ 

particularly when they can be assured that the findings will not be used 

punitively by management. Clinical supervisor's find these data useful in 

monitoring cases and peer reviewers can use the data to^^sure the quality of 

the programs. The center may use the findings in inaervice training programs 

' . I 

to change the ways that clients are assigned to clinicians or to modify 
clinical procedures* 
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* Documentation of client t^rogress through outcome ratings tan also help 
clients. For example, a ^Ijlent^ i^feceiving alcohol rehabilitation services may 
be criticized by his probation officer because of negative reports made by 
the 'c lient ' s family. The therapist, by demonstrating that the client 
progressing well in the program, could modify .the probation officer's ^ 
attitude toward the client. In addition, in clinical management, outcome 

V f 

indicators are used to ''flag'' areas where treatment ife having problems. 



Program Planning A 

Outcome studies assist center management in setting policy and develop- » 

V 

ing program plans.. Outcome information can b^ used 1) to Improve programs^; 

2) \o conserve resources; and 3) to justify programs. ' Several examples of 

the potential programmatic uses of outcome information follow. 

Outcome infomiation can be used to compare the effectiveness of one 

service to another in a very simple way. A center may be able to demonstrate 

that, treating chronically disabled clients in a day care service is just as 

f 



ef feet iye and costs less than the IVipatient utv;it that was ^used in the past. 
Qr, when recidivism rates and outcome Indlo^tors from th6 twt) groups are com- 
pared^ the day care clients may have lower recidivism', rates and higher outcome 
scores than t,hose treated in. the Inpatient unit, thus showing that day care 
is a more effective treatment modality that also Costs less. 

Outcome information may also be used to support a decision to maintain 
a more costly byt more effective service by demonstrating that clients 
improve mor6 and ..have lower recidivism rates than ttfose treat&d In^a less 
effective service. , ( ■ 
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CI Lent sat I'sfact Ion studies pruvidt^ Inforinat ion on .the. client ' s per- 
cepc lont^ot the valiio nnd quality of treatment and mental health programs In 
general. This information is us,eful in' modifying program procedures. In 
addition, such studies yiehl statistical data on the' accessibility and 
acceptability of vServlces for federal and state mental health- agency reports. 
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SUMMARY 



Outcome evaluation assosses the benefits or results of mental health 
services received by clients or communities.. It is- an evaluation approach, 
that compares descriptive data on the mental health status of clients at. 
different points in t^ime. Its purpose is to trigger action by cliwlcians 

* * 

and managers, in planning; programs and managing; clinical services, 

Haj3^ centers defer ^client outcome evaluat ion* until goal-oriented program 
planning, information sj/stejns and quality assurance programs are in place. 
Outcome evaluation ahould be done^ when there is ^ clearly defined need for 
the data, jpesources "^are available, and the uses of, data^ are agreed on by 
clinic^ians and managers, . ^ . . 

Seven methods of evaluating dlitcomes have been described.' Each method 
has tt« advantages and drawbacks which are f-elated to its specificity, cost, 
and potential uses,. Some^o^ the major considerat ions in selecting an 
appropriate outcome measure are its sensitivity to change in client status, 
its reliability, the usefulness of findings, and the costs of Implementation, 

Client outcomes stadi,es,can be conducted continuously or as special • 

i ■ ' ^ , 

" studies. - A minimal level, of contltiuous data callection Includes the rating 

* 

• of clients* level of functioning at predetermined times and the assessment of 

» - ■ . " . 

■ client aatKlsfattlon , with feedback to clinicians and hianagers at regular 

* intervals, v ' ' - 
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Spocicil studfeB t^rpd to be r(iore problem-^orlented * They are usually 



conducted when thefe is a/clearly defined need for information "about a, 
particular service or grouf> of clients. ' , . * 

Outcome evaluation is used to as^st in clinical management and program 

planning. ^Continuously c'bllect^d outcome data help clinicians get an ^ 

V • / 

objective view of * their work wftlTclien t s . * These data also assist clinical 
supervisors in monitoring cases and identifying the need for inservice train- 
ing or changes in clfnical procecjures. Outcome studies assist in settiqg 
policy, developing program plans, and justifying programs* 
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